1. Field of the Invention
The present invention relates to an improvement in a device having a sight line detecting function for use in cameras or the like.
2. Description of the Related Art
There has been proposed various devices for detecting which position an observer is observing on an observed surface, so-called devices for detecting sight line (visual axis), such as eye cameras.
For example, in U.S. Pat. No. 5,486,892 and No. 6,014,524, a parallel light flux from a light source is projected to the anterior ocular segment of the eyeball of an observer, and the sight line of the observer is determined making use of the cornea reflection images by the light reflected from the cornea and image-forming position of the pupil.
Also, U.S. Pat. No. 5,598,248 proposes an optical device (camera) having a sight line detecting device arranged to perform various types of photographing using a sight-line calibration method wherein personal differences in the sight lines of photographers are corrected.
FIG. 25 is a diagram explaining the principle of sight line detection.
In FIG. 25, reference numerals 13a and 13b each denote an infrared-emitting diode (hereinafter abbreviated as IRED) projecting an infrared light. The IREDs 13a and 13b are disposed along the X-axis direction so as to be substantially symmetrical with respect to the optical axis (z-axis) of the light-receiving lens 12, and illuminate a user""s eyeball 17 from the lower sides (positions offset in the y-axis direction). A portion of the illuminating light reflected from the eyeball is converged on a CCD 14 through the light-receiving lens 12. Here, reference numeral 17a denotes the optical axis of the eyeball, 17b denotes cornea, and 17c denotes iris.
FIG. 26A is a schematic diagram showing an eyeball image projected to the CCD 14, and FIG. 26B is a diagram showing an intensity distribution of the signal from the output line of the CCD 14. Hereinafter, descriptions of the sight line detection will be made with reference to FIGS. 25, 26A, and 26B.
The infrared light projected from the IRED 13b illuminates the cornea 17b of the user""s eyeball 17. Herein, the image d reflected from the cornea (virtual image; hereinafter referred to as a xe2x80x9cPurkinje imagexe2x80x9d or a xe2x80x9cP-imagexe2x80x9d), which image d is formed of a portion of the infrared light reflected from the surface of the cornea 17b, is condensed by the light-receiving lens 12, and forms an image at the position dxe2x80x2 on the CCD 14. Likewise, the infrared light projected from the IRED 13a illuminates the cornea 17b of the user""s eyeball 17. Herein, the Purkinje image e formed of a portion of the infrared light reflected from the surface of the cornea 17b, are condensed by the light-receiving lens 12, and forms an image at the position exe2x80x2 on the CCD 14.
The light flux from the end portions a and b of the iris 17c form images of the end portions a and b at the positions axe2x80x2 and bxe2x80x2 on the CCD 14, respectively, through the light-receiving lens 12. When the rotational angle xcex8 of the optical axis of the eyeball 17 with respect to the optical axis of the light-receiving lens 12 is small, letting the x-coordinates of the end portions a and b of the iris 17c be xa and xb, respectively, the coordinate xc of the center position of the pupil 17d is expressed by the following equation.
xc≅(xa+xb)/2 xe2x80x83xe2x80x83(1) 
Also, the x-coordinate of the midpoint between the Purkinje images d and e substantially coincides with the x-coordinate X0 of the center of curvature o of the cornea 17b. 
Therefore, if the x-coordinates of the occurrence positions d and e of the Purkinje images are represented by (Xdxe2x80x2, Xexe2x80x2), and the standard distance from the center of curvature o of the cornea 17b to the center c of the pupil 17d is represented by Loc, the rotational angle xcex8x of the eyeball optical axis 17a of the eyeball 17 satisfies the following relation.
Locxc3x97sin xcex8x≅(Xdxe2x80x2+Xexe2x80x2)/2xe2x88x92xc xe2x80x83xe2x80x83(2) 
Therefore, by detecting the positions of each of the characteristic points of the eyeball 17 (the centers of each of the Purkinje images and the pupil) projected on the CCD 14, the rotational angle xcex8 of the eyeball optical axis 17a of the eyeball 17 can be determined.
The rotational angle xcex8 of the eyeball optical axis 17a are determined based on the above equation (2) as follows:
xcex2xc3x97Locxc3x97sin xcex8x≅{(Xp0xe2x88x92xcex4x)xe2x88x92Xic}xc3x97Ptx xe2x80x83xe2x80x83(3) 
xcex2xc3x97Locxc3x97sin xcex8y≅{(Yp0xe2x88x92xcex4y)xe2x88x92Yic}xc3x97Pty xe2x80x83xe2x80x83(4) 
Here, xcex2 denotes an image-forming magnification determined by the position of the eyeball 17 with resect to the light-receiving lens 12, and is virtually obtained as a function of the distance |Xdxe2x80x2xe2x88x92Xexe2x80x2| between the two Purkinje images.
Also, xcex8x and xcex8y denote the rotational angles of the eyeball optical axis on the z-x plan, and y-z plan, respectively. (Xp0, Yp0) represents the coordinates of the midpoint between the two Purkinje images on the CCD 14, and (Xic, Yic) represents the center coordinates of the pupil. Ptx and Pty denote pixel pitches in the direction of the x-axis and the y-axis, respectively. xcex4x and xcex4y denote correction terms for correcting the coordinates of the midpoint of a Purkinje image. The correction terms include a component for correcting errors occurring by the user""s eyeball being illuminated not by parallel light but by diverging light. Here, xcex4y includes also a component for correcting an offset component occurring by the user""s eyeball being illuminated by diverging light from the lower eyelid.
When calculating the rotational angle (xcex8x, xcex8y) of the optical axis 17a of the user""s eyeball, the user""s gazing point (x, y) on the observed surface is determined by the following expression.
X[mm]=mxc3x97axxc3x97(xcex8x+bx) xe2x80x83xe2x80x83(5) 
Y[mm]=mxc3x97ayxc3x97(xcex8y+by) xe2x80x83xe2x80x83(6) 
Here, the x-axis direction means the horizontal direction with respect to the observer, and the y-axis direction means the vertical direction with respect to the observer. Coefficient m denotes the transformation coefficient for performing a transformation from the rotational angle of the eyeball 17 to the coordinates of the observed surface. Coefficients ax, bx, ay, and by each denote gazing point calibration coefficient, and each correspond to correction coefficient for making the rotational angle of the user""s eyeball coincide with the gazing point on the observed surface.
Now, a calculation method for the correction terms xcex4x and xcex4y for the midpoint position between Purkinje images will be described with reference to FIGS. 27 to 29.
The intersection point between the straight line connecting the IRED 13a and the IRED 13b and the optical axis (z-axis) of the sight line detecting optical system is set to the origin point. The IREDs 13a and 13b are disposed along the X-axis direction so as to be substantially symmetrical with respect to the origin point, and the x-ordinates and y-ordinates thereof are each the same. Let the ordinates of the IRED 13a be (xe2x88x92Sxi, Syi, Szi), the ordinates of the IRED 13b be (Sxi, Syi, Szi), and the ordinates of the center o of curvature of the cornea of the photographer""s eyeball be (Sxc, Syc, Szc). Also, let the center coordinates of the CCD be (Xs, Xy).
The midpoint P0 between the Purkinje images d and e is equivalent to the position of the Purkinje image occurring due to a single IRED disposed at the midpoint between the IREDs 13a and 13b. Since the sight-line calculation equation is based on the coordinates of the midpoint P0 between the two Purkinje images, if the distance from the midpoint (0, Si, Zi) between the IREDs 13a and 13b to the center of curvature o of the cornea is represented by L, L is given by the following expression.
L={Sxc2+(Syixe2x88x92Syc)2+(Szixe2x88x92Szc)2}xe2x80x83xe2x80x83(7) 
The distance K from the surface of the cornea to the occurrence position of the Purkinje image is expressed as follows, using the Abbe""s invariant.
K=Rc(L+Rc)/{2(L+Rc)xe2x88x92Rc}xe2x80x83xe2x80x83(8) 
Also, the shift amount xcex4x of the midpoint P0 between the Purkinje images P1 and P2 in the x-axis direction (the shift amount in the CCD coordinate system; xcex94x in the eyeball coordinate system) satisfies the following relationship.
L/(Kxe2x88x92Rc)={(Xp0xe2x88x92Xs)+xcex4x}/xcex4x xe2x80x83xe2x80x83(9) 
When developing the above equation, the shift amount xcex4x is calculated as
xcex4x=Rc(Xp0xe2x88x92Xs)/{2(L+Rc)}xe2x80x83xe2x80x83(10) 
Likewise, the shift amount xcex4y of the midpoint P0 between the Purkinje images P1 and P2 in the y-axis direction satisfies the following relationship.
L/(Kxe2x88x92Rc)={(Xp0xe2x88x92Ys)+Syi(xcex2p/Pty)+xcex4y}/xcex4y xe2x80x83xe2x80x83(11) 
When developing the above equation, the shift amount xcex4y is calculated as
xcex4y=Rc{(Yp0xe2x88x92Ys)+Syi(xcex2p/Pty)}/{2(L+Rc)}xe2x80x83xe2x80x83(12) 
Here, if we define the image-forming magnification xcex2 of the sight line detecting optical system as a secondary function of the distance Sz from the exit surface (origin point) of the eye piece lens, xcex2 is given by the following expression.
xcex2=b1xc3x97Sz2+b2xc3x97Sz+b3 xe2x80x83xe2x80x83(13) 
Here, coefficients b1 and b3 are calculated by firstly obtaining the image-forming magnification at a predetermined distance by an optical simulation, and then by secondarily approximating the obtained values.
Letting the distance from the apex of the cornea in the z-axis direction be Szp, the image-forming magnification
xcex2p is calculated as
xcex2p=b1(xe2x88x92Szpxe2x88x92Szc)2+b2(xe2x88x92Szpxe2x88x92Szc)+b3 xe2x80x83xe2x80x83(14) 
Hereinafter, in order to determine Szp and Szc, a plurality of times of calculation routines are executed so as to make Szp and Szc converge.
From the comparison between the equation (9) with the equation (11), it is noticed that the equation (11) includes an additional term xe2x80x9cSyi(xcex2p/Pty)xe2x80x9d. The reason why this additional term has occurred, is because, as shown in FIG. 29, an IRED does not exist along the y-axis direction across the z-axis with respect to the IREDs. 13a and 13b unlike the case in FIG. 28, and hence, a Purkinje image does not exist. The shift amount xcex4y in the y-axis direction, therefore, cannot be obtained directly from the midpoint P0 between the Purkinje images P1 and P2.
Therefore, in the shift amount xcex4y in the y-axis direction, apart from errors in the above-described secondary approximation of the coefficients b1 to b3, other approximate calculation errors in the convergence calculation, and quantization errors in a microcomputer are added with respect to the shift amount xcex4x in the x-axis direction, with the result that the shift amount xcex4y in the y-axis direction is interior in the accuracy to xcex4x. Therefore, in the rotational angle (xcex8x, xcex8y) of the optical axis 17a of the user""s eyeball, which rotational angle is determined by the equation (3) including xcex8x and the equation (4) including xcex8y, xcex8y has a defect of being interior in the accuracy to xcex8x.
Once the rotational angle xcex8x and xcex8y of the photographer"" eyeball has been calculated from the equation (3) including xcex8x and the equation (4) including xcex8y, correction coefficients ax, bx, ay, and by for personal differences are obtained from the above equations (4) and (5), and thereby the coordinates on a focusing plate are calculated.
Here, as possible means for avoiding the defect of xcex8y being interior in the accuracy to xcex8x, there is a method wherein an IRED 13e provided at a position along the y-axis direction across the z-axis with respect to the IREDs 13a and 13b, is used for illumination to obtain a new Purkinje image, and wherein, by obtaining xcex8y in the same manner as in the case of xcex8x, xcex8y is provided with the same accuracy as that of xcex8x. In short, the IRED 13e is arranged so as not to be on the same straight line with the IREDs 13a and 13b.
This will be elucidated with reference to FIGS. 30 and 31 wherein the illumination of the IRED 13e has been added.
As in the case of FIG. 27A, FIG. 30 shows the positions of the characteristic points of the eyeball 17 (the centers of each of the Purkinje images and the pupil) projected on the CCD 14. Herein, a new Purkinje image 3 exists at the coordinates (Xexe2x80x3, Yexe2x80x3), the coordinates of the conventional Purkinje images P1 and P2 corresponding to the IREDs 13a and 13b exist at (Xexe2x80x2, Yexe2x80x2) and (Xdxe2x80x2, Ydxe2x80x2), respectively, and the midpoint coordinates (Xp1, Yp1) of the Purkinje images P1 and P3 which are arranged parallel to the y-coordinate, constitute a P-image center in the y-axis direction.
FIG. 31 is representative of the above-described explanation, using a y-x plan eyeball coordinate diagram as in the case of FIG. 29.
The above equation (4) can be represented as
xcex2xc3x97Locxc3x97sin xcex8y≅{(Yp1xe2x88x92xcex4y)xe2x88x92Yic}xc3x97Pty xe2x80x83xe2x80x83(15) 
The correction term xcex4y can be expressed as
xcex4y={Rc(Yp1xe2x88x92Ys)}/2{(L+Rc)}xe2x80x83xe2x80x83(16) 
This indicates that the shift amount xcex4y in the y-axis direction can be expressed by an equation similar to the equation (10) giving the shift amount xcex4x in the x-axis direction.
Meanwhile, U.S. Pat. No. 5,552,854 also discloses a device wherein three illumination light sources are arranged. In this disclosure, however, among the three light sources, two light sources are selected so as not to put the center of curvature of the cornea therebetween, and an Purkinje image is obtained to calculate the sight line.
When detecting the sight line of an observer, a Purkinje image corresponding to illumination light sources is not necessarily obtained. That is because, particularly in the case of a Purkinje image on the upside corresponding to the IRED 13e or 13b on the inside, the illumination light is blocked by the upper eyelid and eyelash of the observer, depending on observation conditions and personal differences.
Accordingly, there is a proposal to add an IRED 13f, which is provided along the y-axis across the z-axis with respect to the IRED 13a and 13b, and to use it for illumination. By doing so, a probability increases that at least one of the Purkinje images corresponding to the IREDs 13e and 13f will be achieved, even if the illumination light source is blocked by the upper eyelid and eyelash of the observer, depending on observation conditions and personal differences.
When at least three Purkinje images have been obtained corresponding to the IREDs 13a, 13b, 13e, and 13f, the accuracy of the eyeball rotational angles xcex8x and xcex8y can be made equal to each other, by properly providing the correction coefficients ax, bx, ay, and by for personal differences, as described above.
FIG. 32 shows the positions of the characteristic points (the centers of each of the Purkinje images and the pupil) of the eyeball 17 projected on the CCD 14 when the four Purkinje images have been obtained, as in the cases of FIGS. 27 and 30.
When the four Purkinje images corresponding to the IREDs 13a, 13b, 13e, and 13f, each of the center positions of the Purkinje images in the x-axis and y-axis directions is calculated from not only the coordinates of two Purkinje images, but also from those of the four Purkinje images, with respect to FIGS. 28 to 31. Therefore, by properly providing the correction coefficients ax, bx, ay, and by for personal differences, the accuracy of the eyeball rotational angles can be more improved not only for xcex8y, but also for xcex8x.
However, even if the new illumination light sources IRED 13e and 13f are thus added, the Purkinje image corresponding to one of these Purkinje images is not necessarily obtained. There is a possibility that neither of the two Purkinje images is obtained. When no Purkinje image corresponding to the IRED 13e and 13f is obtained, but only the Purkinje images corresponding to the IRED 13a and 13b are obtained, xcex4y can be obtained from the equation (12). On the other hand, when a Purkinje image corresponding to one of the IRED 13e and 13f is obtained, xcex4y can be obtained from the equation (16). Therefore, even if the four illumination light sources IREDs 13a to 13d are employed, each of correction coefficients ay and by may change into a different value, depending on whether Purkinje images by the IREDs 13e and 13f can be obtained or not. This raises a possibility that the Purkinje images obtained during a sight line detecting operation do not appropriately correspond to those obtained during the calculation of calibration coefficients.
It is, therefore, necessary to ensure a sight line detecting function which can always perform reliable sight line detection, irrespective of whether Purkinje images by the IREDs 13e and 13f can be obtained or not. In this respect, the sight line detecting function still has a room for improvement.
In accordance with one aspect of the present invention, by obtaining a cornea reflection image from an image pickup member which receives reflected light from an eyeball, correction data are calculated for correcting the error between the eyeball rotational angle which is formed by the rotation of an observer""s eyeball, and the observer""s sight line. With respect to the same observer, a plurality of the correction data corresponding to the number and the configuration of cornea reflection images is calculated. Thereby, even when a portion of cornea reflection images cannot be obtained, sight line detection can be performed using the correction data corresponding to the number and the configuration of the cornea reflection images which are obtained at that time.
In accordance with another aspect of the present invention, from the cornea reflection images which were simultaneously detected, a plurality of correction data corresponding to different numbers of cornea reflection images is calculated. For example, when three cornea reflection images are detected, the correction data corresponding to three cornea reflection images, and the correction data corresponding to two cornea reflection images, are calculated. This saves us time and labor of detecting cornea reflection images over a plurality of times to calculate a plurality of correction data corresponding the number of the cornea reflection images.